View Issue Details

IDProjectCategoryView StatusLast Update
0001407T99X171.00 SKB EagleSW Issuepublic2022-02-23 10:42
Reporter(ALTech) Younkwang Jung Assigned To(ALTech) Younkwang Jung Due Date
PrioritynormalSeveritys4-minorReproducibilityhave not tried
Status closedResolutionnot fixable 
Summary0001407: [Smart3][BTF][VoC][Tombstone] com.android.bluetooth
DescriptionHi kerwin

We are checking a lot of Ramoops cases.
Among them, there was a case of tombstone in com.android.bluetooth, which includes logcat information.

====================================================================================================
Build fingerprint: 'SKB/BFX_AT100/BFX-AT100:10/QTT5.200429.010/15.531.10-20210917:user/release-keys'
Revision: '0'
ABI: 'arm'
Timestamp: 2021-12-03 18:19:48+0900
pid: 7973, tid: 7998, name: alarm_default_c >>> com.android.bluetooth <<<
uid: 1002
signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
    r0 00000000 r1 00001f3e r2 00000006 r3 ca63f978
    r4 ca63f98c r5 ca63f970 r6 00001f25 r7 0000016b
    r8 ca63f988 r9 ca63f978 r10 ca63f9a8 r11 ca63f998
    ip 00001f3e sp ca63f948 lr f056dc43 pc f056dc56

backtrace:
      #00 pc 0005ec56 /apex/com.android.runtime/lib/bionic/libc.so (abort+166) (BuildId: 4a0757f231650c15e76a2251099242e8)
      #01 pc 00180ea3 /system/lib/libbluetooth.so (startup_timer_expired(void*)+50) (BuildId: effbd3fa1ca83e1078c483c0be79ca2a)
      #02 pc 00212b77 /system/lib/libbluetooth.so (alarm_ready_generic(alarm_t*, std::__1::unique_lock<std::__1::mutex>&)+326) (BuildId: effbd3fa1ca83e1078c483c0be79ca2a)
      #03 pc 002129ff /system/lib/libbluetooth.so (alarm_queue_ready(fixed_queue_t*, void*)+66) (BuildId: effbd3fa1ca83e1078c483c0be79ca2a)
      #04 pc 00215d5d /system/lib/libbluetooth.so (internal_dequeue_ready(void*)+48) (BuildId: effbd3fa1ca83e1078c483c0be79ca2a)
      #05 pc 002169ad /system/lib/libbluetooth.so (run_reactor(reactor_t*, int)+264) (BuildId: effbd3fa1ca83e1078c483c0be79ca2a)
      #06 pc 00216885 /system/lib/libbluetooth.so (reactor_start(reactor_t*)+44) (BuildId: effbd3fa1ca83e1078c483c0be79ca2a)
      #07 pc 0021799d /system/lib/libbluetooth.so (run_thread(void*)+148) (BuildId: effbd3fa1ca83e1078c483c0be79ca2a)
      #08 pc 000a6087 /apex/com.android.runtime/lib/bionic/libc.so (__pthread_start(void*)+20) (BuildId: 4a0757f231650c15e76a2251099242e8)
      #09 pc 00060141 /apex/com.android.runtime/lib/bionic/libc.so (__start_thread+30) (BuildId: 4a0757f231650c15e76a2251099242e8)
========================================================================================================================

logcat information ( main.log )
=============================================================
12-03 18:19:49.911 7973 7994 E AdapterState: BLE_TURNING_ON : BLE_START_TIMEOUT
12-03 18:19:49.911 7973 7994 I AdapterState: BLE_TURNING_OFF : entered
12-03 18:19:49.922 7973 7973 W : [1203/181949.922530:WARNING:bta_gattc_api.cc(61)] GATTC Module not enabled/already disabled
12-03 18:19:49.922 7973 7973 W : [1203/181949.922712:WARNING:bta_gatts_api.cc(56)] GATTS Module not enabled/already disabled
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: exception when invoking removeOnUidImportanceListener
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: java.lang.IllegalArgumentException: Listener not registered: com.android.bluetooth.gatt.ScanManager$2@21e5145
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at android.app.ActivityManager.removeOnUidImportanceListener(ActivityManager.java:3382)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at com.android.bluetooth.gatt.ScanManager.cleanup(ScanManager.java:157)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at com.android.bluetooth.gatt.GattService.cleanup(GattService.java:266)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at com.android.bluetooth.btservice.ProfileService.onDestroy(ProfileService.java:211)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at android.app.ActivityThread.handleStopService(ActivityThread.java:4116)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at android.app.ActivityThread.access$1900(ActivityThread.java:219)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1896)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at android.os.Handler.dispatchMessage(Handler.java:107)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at android.os.Looper.loop(Looper.java:214)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at android.app.ActivityThread.main(ActivityThread.java:7356)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at java.lang.reflect.Method.invoke(Native Method)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
12-03 18:19:49.923 7973 7973 W BtGatt.ScanManager: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:930)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: exception when invoking unregisterReceiver(mLocationReceiver)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: java.lang.IllegalArgumentException: Receiver not registered: com.android.bluetooth.gatt.ScanManager$3@409d09a
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at android.app.LoadedApk.forgetReceiverDispatcher(LoadedApk.java:1429)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at android.app.ContextImpl.unregisterReceiver(ContextImpl.java:1543)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at android.content.ContextWrapper.unregisterReceiver(ContextWrapper.java:664)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at com.android.bluetooth.gatt.ScanManager.cleanup(ScanManager.java:178)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at com.android.bluetooth.gatt.GattService.cleanup(GattService.java:266)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at com.android.bluetooth.btservice.ProfileService.onDestroy(ProfileService.java:211)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at android.app.ActivityThread.handleStopService(ActivityThread.java:4116)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at android.app.ActivityThread.access$1900(ActivityThread.java:219)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1896)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at android.os.Handler.dispatchMessage(Handler.java:107)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at android.os.Looper.loop(Looper.java:214)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at android.app.ActivityThread.main(ActivityThread.java:7356)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at java.lang.reflect.Method.invoke(Native Method)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
12-03 18:19:49.924 7973 7973 W BtGatt.ScanManager: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:930)
12-03 18:19:50.104 3544 3754 I BOXKEY-STDVB: [vendor/skb/framework/btf_dev/interface/btvservice/1.0/default/BtvMediaPlayer.cpp][invoke] called device [0]
12-03 18:19:50.105 4718 4718 I IPTV : BTF|getCurrentPosition||1359|Get Current Position from Main 85281
12-03 18:19:50.105 4718 4718 I IPTV : BTF|getDuration||1331|Get Duration from Main 4843839
12-03 18:19:50.106 4718 4718 I VODPlayer: onPlayerTimeInfo() [85.281s/4843.839s]
12-03 18:19:50.156 3620 3620 E /system/bin/tombstoned: Tombstone written to: /data/tombstones/tombstone_00
=============================================================

Please analyze the cause of this tombstone.

Thank you
YK.jung
TagsNo tags attached.
Attach Tags

Users monitoring this issue

User List (ALTech) JunGyu Kim , (ALTech) SY Yoon , (SW) Brent Choi

Activities

(ALTech) Younkwang Jung

2021-12-08 10:30

developer  

(SW) Kerwin Chen

2021-12-08 15:46

developer   ~0008872

Hi YK,

How many Smart3 have such kind of log ?
It looks like BT stack can't receive response from BT chip.

Those tombstones were generated because of timer expired in BT stack.
BT stack will check if BT chip works or not by the timer.
You can check part of source code as below

in system/bt/hci/src/hci_layer.cc
...
alarm_set(startup_timer, startup_timeout_ms, startup_timer_expired, NULL);
...
static void startup_timer_expired(UNUSED_ATTR void* context) {
  LOG_ERROR(LOG_TAG, "%s", __func__);
  LOG_EVENT_INT(BT_HCI_TIMEOUT_TAG_NUM, HCI_STARTUP_TIMED_OUT);
  hci_close();
  if (abort_timer.IsScheduled()) {
    LOG_ERROR(LOG_TAG, "%s: waiting for abort_timer", __func__);
    return;
  }
  abort(); <== tombstone will be generated
}

Regards,
Kerwin

(ALTech) Younkwang Jung

2021-12-08 16:53

developer   ~0008874

Hi Kerwin

There were about 337 cases of this tombstone.

I have one question.
Do you know why there is a code(abort()) that causes a tombstone?

Thank you
YK.Jung

(SW) Kerwin Chen

2021-12-08 18:05

developer   ~0008876

Hi YK,

That abort() is added in Android BT stack for long time.
I don't know why Google add it here.

>>>> "There are about 337 cases of this tombstone"
Do you mean there are 337 tombstone files or the tombstone files are found on different Smart3 box ?

Regards,
Kerwin

(ALTech) Younkwang Jung

2021-12-08 22:25

developer   ~0008879

Hi Kerwin

Thank you for your quick answer.

I uploaded the Ramoops files on the path below.
http://altserver01.iptime.org/release_by_ALT/Ramoops/

Approximately 300 Ramoops files are collected a day.
Among them, there are 10 to 15 cases of this tombstone.
In total(337 times), this tombstone is not a small number of times.
and Some STB may have 1 or 2 of this tombstone.
The VoC log attached here was the only case where tombstone and logcat overlap.

Thank you
YK.jung

(ALTech) Younkwang Jung

2021-12-21 07:52

developer   ~0008963

Hi kerwin
FW : v15.532.20
Please check the attached files as well.
Is there any way to debug this situation?
For example, adding logs.
We have to debug about this issue.

Thank you
YK.Jung

(SW) Kerwin Chen

2021-12-21 13:56

developer   ~0008965

Hi YK,

Do you know the previous version on these Smart3 boxes ?
Does the issue happen before on these boxes ?

Thank you ~

(ALTech) Younkwang Jung

2021-12-21 17:37

developer   ~0008968

Hi Kerwin

According to previous records,
This issue also occurred about 330 times in the previous version (v15.531.10)

Thank you
YK.Jung

(SW) Kerwin Chen

2021-12-24 10:58

developer   ~0009106

Hi YK,

Does tombstone happen on dedicated STB only ?
Since the tombstone is seen in older version, I suggest to get 1 Smart3 box back for analysis.

Thanks !

(ALTech) Younkwang Jung

2021-12-24 11:34

developer   ~0009109

Hi Kerwin

50 tombstones were generated in 7 STBs in a week.
It is occurring in a specific box, so I requested SKB to collect STB and RCU for analysis.
but there is no feedback yet.

Thanks
YK.Jung

(ALTech) Younkwang Jung

2022-01-11 12:22

developer   ~0009317

Last edited: 2022-01-11 12:23

View 2 revisions

Hi Kerwin

Approximately 83 tombstones occurred in 8 STBs ( per day )
I have a VoC meeting at tomorrow.
Is there anything else we can do about this?

Thank you
YK.Jung

bt_tombstone.zip (10,810,367 bytes)

(SW) Kerwin Chen

2022-01-11 18:54

developer   ~0009325

Hi YK,

Per update in "https://mantis.cnsbg.foxconn.com/vaas/view.php?id=1407#c8872"
It looks there is something wrong on those STBs.
I think we should get those STBs back and then do analysis.

Thank you ~

(SW) Kerwin Chen

2022-02-22 14:46

developer   ~0009487

HW damage

Issue History

Date Modified Username Field Change
2021-12-08 10:30 (ALTech) Younkwang Jung New Issue
2021-12-08 10:30 (ALTech) Younkwang Jung Status new => assigned
2021-12-08 10:30 (ALTech) Younkwang Jung Assigned To => (SW) Kerwin Chen
2021-12-08 10:30 (ALTech) Younkwang Jung File Added: ME0041_20211206_40_23_43_e0_6b_ea.zip
2021-12-08 10:31 (ALTech) Younkwang Jung Issue Monitored: (SW) Brent Choi
2021-12-08 10:31 (ALTech) Younkwang Jung Issue Monitored: (ALTech) SY Yoon
2021-12-08 10:31 (ALTech) Younkwang Jung Issue Monitored: (ALTech) JunGyu Kim
2021-12-08 15:46 (SW) Kerwin Chen Note Added: 0008872
2021-12-08 15:47 (SW) Kerwin Chen Assigned To (SW) Kerwin Chen => (ALTech) Younkwang Jung
2021-12-08 15:47 (SW) Kerwin Chen Status assigned => feedback
2021-12-08 16:53 (ALTech) Younkwang Jung Note Added: 0008874
2021-12-08 18:05 (SW) Kerwin Chen Note Added: 0008876
2021-12-08 22:25 (ALTech) Younkwang Jung Note Added: 0008879
2021-12-21 07:52 (ALTech) Younkwang Jung File Added: ME0041_20211217_40_23_43_cd_0d_4b.zip
2021-12-21 07:52 (ALTech) Younkwang Jung File Added: ME0041_20211217_40_23_43_e0_6b_ea (1).zip
2021-12-21 07:52 (ALTech) Younkwang Jung File Added: ME0041_20211217_40_23_43_e0_6b_ea.zip
2021-12-21 07:52 (ALTech) Younkwang Jung File Added: ME0041_20211217_28_cd_c4_fb_79_f0.zip
2021-12-21 07:52 (ALTech) Younkwang Jung File Added: ME0041_20211217_40_5b_d8_e2_f6_44 (1).zip
2021-12-21 07:52 (ALTech) Younkwang Jung File Added: ME0041_20211217_40_5b_d8_e2_f6_44.zip
2021-12-21 07:52 (ALTech) Younkwang Jung Note Added: 0008963
2021-12-21 13:56 (SW) Kerwin Chen Note Added: 0008965
2021-12-21 17:37 (ALTech) Younkwang Jung Note Added: 0008968
2021-12-23 17:47 (SW) Kerwin Chen Severity s2-severe => s3-moderate
2021-12-23 17:47 (SW) Kerwin Chen Description Updated View Revisions
2021-12-24 09:18 (SW) Kerwin Chen Severity s3-moderate => s4-minor
2021-12-24 10:58 (SW) Kerwin Chen Note Added: 0009106
2021-12-24 11:34 (ALTech) Younkwang Jung Note Added: 0009109
2022-01-11 12:22 (ALTech) Younkwang Jung File Added: bt_tombstone.zip
2022-01-11 12:22 (ALTech) Younkwang Jung Note Added: 0009317
2022-01-11 12:23 (ALTech) Younkwang Jung Note Edited: 0009317 View Revisions
2022-01-11 18:54 (SW) Kerwin Chen Note Added: 0009325
2022-02-22 14:46 (SW) Kerwin Chen Status feedback => resolved
2022-02-22 14:46 (SW) Kerwin Chen Resolution open => not fixable
2022-02-22 14:46 (SW) Kerwin Chen Note Added: 0009487
2022-02-23 10:42 (SW) Kerwin Chen Status resolved => closed